Environmental Sensor Study: Melbourne
Authored by: Julian Cape
Duration: 60 mins
Level: Beginner
Pre-requisite Skills: Python, Data Engineering/Analysis
Scenario

As a city planner I want to determine the ideal number of environmental sensors for the city of Melbourne to help it reach its smart city goals. How many should be installed?

What this use case will teach you

At the end of this use case you will:

  • understand how to access and collect data from two different APIs.
  • have learnt how to work with and manipulate pandas and geopandas dataframes.
  • have explored a variety of datasets relevant to the location of environmental sensors.
  • have combined and visualised multiple datasets over a map of the city.
  • have connected to the Environmental Protection Agencies environmental sensor API.
  • calculated the ideal number of environmental sensors and their position around the city.
Why an environmental sensor network?

Issue being addressed

In June 2021 the Melbourne City Council endorsed its Economic Development Strategy 2031. This strategy is a plan for economic, social, and cultural recovery for the city in the ten years following the Covid-19 pandemic. The strategy introduces eight key priorities covering business, housing, health, technology, the environment, and the arts.

Priority 7 is to become a digitally connected city. This priority concerns adapting to the rapidly evolving technological landscape and becoming a connected, knowledge-enabled smart city. A smart city is one which uses a range of electronic methods and sensors to collect data and inform decision making.

In Trimester 1, 2022 a data-driven study to determine the ideal locations for Green Walls around the City of Melbourne was conducted by myself and Ryan Waites on behalf of the City. One of the key findings resulting from the study was that the amount of environmental sensors currently located around the city is not enough to accurately inform decision making. This led to the ideation of this study, one to determine the ideal number of environmental sensors for the City of Melbourne.

The idea of a smart city arose in the 1990s, but it was only in recent times that technology has advanced to the point where the concept can become a reality. Sensors are becoming ever smaller and cheaper, and computers are now powerful enough to process vast quanitites of data. Artificial Intelligence algorithms are able to analyse this data in ways that is impossible for humans, revealing insights and connections previously unthought of. The aim of this study is to support the installation of an extensive environmental sensor network that provides actionable data now and supports Melbourne's smart city data ecosystem in the future.

Issues with current air quality standards

One of the Australian Standards (AS 3580.1.1) for siting air monitoring stations specifies that they should be located more than 50 metres away from a road. I hope you agree with me dear reader that this is counterintuitive to the purpose of conducting air monitoring. To discover the effects of air quality on public health we should aim to establish the level of exposure people are having to various pollutants on a daily basis. The only time that someone outdoors in the Melbourne CBD is more than 50m away from a road is when they are in a park. Most people outdoors are not in parks. Many people enjoy outdoor dining at locations less than one metre away from a road on a daily basis!

Variable environmental factors like temperature and wind have an effect on the movement of gases. An extensive sensor network would provide ongoing valuable scientific data relating to a range of activities taking place in Melbourne. The scale of chemical and physical reactions affects how they proceed. The data from an extensive sensor network could be analysed against a range of other datasets from things such an rainfall to solar energy. As yet unknown causal relationships may be discovered.

Datasets overview

All of the datasets used in this analysis come from the City of Melbourne's open data portal, with the exception of the data taken from the Environment Protection Authorities Environmental Monitoring API.

The first dataset to be analysed is that of the city's Microclimate sensors, to determine the extent of the current network. We will access this information (Microclimate Sensor Locations) and map the results. Let's get into it!

Package and data imports

To begin we will import the required libraries and datasets to perform our exploratory analysis and visualisation of the datasets.

The following are core packages required for this exercise:

  • The sodapy package is required for accessing open data from SOCRATA compliant open data websites.
  • The folium package is required for the mapping visualisations
  • Geopandas is used to manipulate some datasets and create related spatial information

If you attempt to run this first cell and there is a 'module not found' error, you may need to install the package on your system. Try running: pip install -package name- to install the missing package. If this doesn't work, you may need to try Google!

In [ ]:
#This notebook was created using an ipython notebook with Jupyter Lab. 
# Depending on how you are running this notebook you may need some of the following packages. Try running these if some parts of the notebook don't work for you!

# !pip install sodapy
# !pip install geopandas
# !pip install rtree
# !pip install pygeos
# !pip install geopy

#Issue with using folium through Jupyter notebook was resolved by downgrading markupsafe as follows: 
# !pip install markupsafe==2.0.1
In [ ]:
#File manipulation
import os
from datetime import datetime
from sodapy import Socrata

#Data manipulation
import numpy as np
import pandas as pd
import geopandas as gpd
import seaborn as sns
from shapely.geometry import polygon, shape, point
from shapely import geometry


#Visualisation
import matplotlib.pyplot as plt
import branca.colormap as cm
import folium
from folium.plugins import MarkerCluster
In [ ]:
#Details for City of Melbourne data on Socrata website.

apptoken = os.environ.get("bajmvQjws2C8yfqmVSkOtOU9L") # App token created on Socrata website
domain = "data.melbourne.vic.gov.au"
client = Socrata(domain, apptoken) # Open Dataset connection
Current Sensor Network

Let's first import the sensor data from Melbourne Open Data and look at the number and location of the network the city currently has in place.

Sensor location dataset: https://data.melbourne.vic.gov.au/Environment/Microclimate-Sensor-Locations/irqv-hjr4

In [ ]:
#Loading Melbourne Microclimate Sensor Location data as a Geopandas Dataframe

dataresource = client.get('irqv-hjr4')

# Retrieve data from Microclimate Sensor Locations dataset. 
sensor_locations = gpd.GeoDataFrame.from_dict(dataresource)
sensor_locations
Out[ ]:
site_id gateway_hub_id last_data site_status start_reading longitude latitude location end_reading
0 1011 arc1055 2022-11-29T17:00:01.000 C 2021-07-07T00:00:00.000 144.952222 -37.822222 {'type': 'Point', 'coordinates': [144.952222, ... NaN
1 1004 arc1048 NaN R 2019-11-15T00:00:00.000 144.964635 -37.800575 {'type': 'Point', 'coordinates': [144.964635, ... 2021-06-13T00:00:00.000
2 1005 arc1050 NaN R 2019-11-15T00:00:00.000 144.965052 -37.800629 {'type': 'Point', 'coordinates': [144.965052, ... 2021-06-13T00:00:00.000
3 1006 arc1112 NaN R 2021-05-20T00:00:00.000 144.952065 -37.822486 {'type': 'Point', 'coordinates': [144.952065, ... 2021-06-28T00:00:00.000
4 1002 arc1046 NaN R 2019-11-15T00:00:00.000 144.964122 -37.800524 {'type': 'Point', 'coordinates': [144.964122, ... 2021-06-13T00:00:00.000
5 1010 arc1112 2022-11-29T17:00:01.000 C 2021-06-29T00:00:00.000 144.95222195 -37.82250002 {'type': 'Point', 'coordinates': [144.95222195... NaN
6 1009 arc1050 2022-11-29T17:00:01.000 C 2021-06-14T00:00:00.000 144.96570467 -37.81686763 {'type': 'Point', 'coordinates': [144.96570467... NaN
7 1008 arc1045 NaN R 2021-06-14T00:00:00.000 144.96705703 -37.81746522 {'type': 'Point', 'coordinates': [144.96705703... 2021-06-20T00:00:00.000
8 1014 arc1045 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.967222 -37.8175 {'type': 'Point', 'coordinates': [144.967222, ... NaN
9 1001 arc1045 NaN R 2019-11-15T00:00:00.000 144.966492 -37.800793 {'type': 'Point', 'coordinates': [144.966492, ... 2021-06-13T00:00:00.000
10 1013 arc1047 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.956389 -37.811944 {'type': 'Point', 'coordinates': [144.956389, ... NaN
11 1003 arc1047 NaN R 2019-11-15T00:00:00.000 144.960923 -37.8023 {'type': 'Point', 'coordinates': [144.960923, ... 2021-06-13T00:00:00.000
12 1007 arc1113 2022-07-20T15:00:01.000 C 2021-05-20T00:00:00.000 144.951835 -37.82246 {'type': 'Point', 'coordinates': [144.951835, ... NaN
13 1012 arc1048 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.97 -37.813333 {'type': 'Point', 'coordinates': [144.97, -37.... NaN
14 1015 arc1046 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.9725 -37.810278 {'type': 'Point', 'coordinates': [144.9725, -3... NaN
15 1016 arc1049 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.960556 -37.812778 {'type': 'Point', 'coordinates': [144.960556, ... NaN

The Environmental Protection Agency (EPA) also has an air monitoring station in the City of Melbourne. On their website they have instructions on how to connect to their Environment Monitoring API.

We need to make an http request to the API containing an identification key (X-API-Key) and the Melbourne CBD monitoring station site id taken from the EPA website (4afe6adc-cbac-4bf1-afbe-ff98d59564f9).

For completeness we'll get the details of this station and add it to our current network database.

In [ ]:
########### Python 3.2 #############
import http.client, urllib.request, urllib.parse, urllib.error, base64

headers = {
    # Request headers
    'X-TransactionID': '',
    'X-TrackingID': '',
    'X-SessionID': '',
    'X-CreationTime': '',
    'X-InitialSystem': '',
    'X-InitialComponent': '',
    'X-InitialOperation': '',
    'X-API-Key': '4d234668273941a4ac2867c2dda06c2e',              #API Key is for a personal account created on the EPA website. As no sensitive information is involved it is ok to share here. 
}

params = urllib.parse.urlencode({
    # Request parameters
    'since': '',
    'until': '',
    'interval': '',
})

try:
    conn = http.client.HTTPSConnection('gateway.api.epa.vic.gov.au')
    conn.request("GET", "/environmentMonitoring/v1/sites/4afe6adc-cbac-4bf1-afbe-ff98d59564f9/parameters?%s" % params, "", headers)
    response = conn.getresponse()
    data = response.read()
    print(data)
    conn.close()
except Exception as e:
    print("[Errno {0}] {1}".format(e.errno, e.strerror))

####################################

The API call was successful, but the information isn't in a very readable format. It looks like it could be a JSON file. Let's pass it to python's json library and see if it makes it more readable.
In [ ]:
import json

jsonResponse = json.loads(data)
jsonResponse

This JSON has the coordinates of the station. Let's add it's information to our sensor dataframe.

The following cell shows how to add a row of information to a dataframe. We're inserting null for most of the values as they aren't important for what we're doing, which is determining sensor locations.

In [ ]:
sensor_locations.loc[len(sensor_locations.index)] = ['EPA', 'Null', 'Null', 'Null', 'Null', 144.97, -37.8073959, "{'type': 'Point', 'coordinates': [-37.8073959, 144.97]}", 'Null']

Next we'll reverse geocode the lat and long of the sensor locations to append their address to the sensors dataframe. We'll use the reverse_geocode method that comes with geopandas. It takes in coordinates and outputs an address.

In [ ]:
sensor_locations['Address'] = " "  

#This creates an empty column called 'Address' at the end of the dataframe. 

#Creating an empty column in the dataframe allows it to be iterated over using the iterrows method.
In [ ]:
from shapely.geometry import Point

#The following code iterates over the rows in the dataframe, calling the reverse geocode method on each row and adding the resulting information to the 'address' column.
for index, row in sensor_locations.iterrows():
    sensor_locations['Address'].iloc[[index]] = gpd.tools.reverse_geocode([Point(float(row['longitude']), float(row['latitude']))])['address']
In [ ]:
# pd.set_option('display.max_colwidth', None)    #Setting this value ensures the addresses aren't abbreviated in the notebook, enabling them to be read. 

print(sensor_locations['Address'])

0     Melbourne Convention Centre Parking, Wurundjer...
1     Swanston Street/Grattan Street, Bella Lane, 30...
2     Rakuzen, Grattan Street, 3053, Grattan Street,...
3     Melbourne Convention Centre Parking, Wurundjer...
4     683, Swanston Street, 3053, Swanston Street, M...
5     Siddeley Street, 3008, Melbourne, Victoria, Au...
6     Crumpler, 40-44, Degraves Street, 3000, Degrav...
7     Stop 5: Flinders Street Station, Swanston Stre...
8     Flinders Street, 3000, Melbourne, Victoria, Au...
9     Lygon Street/Grattan Street, Grattan Street, 3...
10    William Street, 3000, Melbourne, Victoria, Aus...
11    Corkman Park (formerly The Corkman Irish Pub),...
12    Siddeley Street, 3008, Melbourne, Victoria, Au...
13    Bank Australia, 104, Little Collins Street, 30...
14    2, Little Bourke Street, 3000, Little Bourke S...
15    Hardware Lane/Lonsdale Street, Lonsdale Street...
16    Victoria Street, 3000, Melbourne, Victoria, Au...
Name: Address, dtype: object
In [ ]:
sensor_locations
Out[ ]:
site_id gateway_hub_id last_data site_status start_reading longitude latitude location end_reading Address
0 1011 arc1055 2022-11-29T17:00:01.000 C 2021-07-07T00:00:00.000 144.952222 -37.822222 {'type': 'Point', 'coordinates': [144.952222, ... NaN Melbourne Convention Centre Parking, Wurundjer...
1 1004 arc1048 NaN R 2019-11-15T00:00:00.000 144.964635 -37.800575 {'type': 'Point', 'coordinates': [144.964635, ... 2021-06-13T00:00:00.000 Swanston Street/Grattan Street, Bella Lane, 30...
2 1005 arc1050 NaN R 2019-11-15T00:00:00.000 144.965052 -37.800629 {'type': 'Point', 'coordinates': [144.965052, ... 2021-06-13T00:00:00.000 Rakuzen, Grattan Street, 3053, Grattan Street,...
3 1006 arc1112 NaN R 2021-05-20T00:00:00.000 144.952065 -37.822486 {'type': 'Point', 'coordinates': [144.952065, ... 2021-06-28T00:00:00.000 Melbourne Convention Centre Parking, Wurundjer...
4 1002 arc1046 NaN R 2019-11-15T00:00:00.000 144.964122 -37.800524 {'type': 'Point', 'coordinates': [144.964122, ... 2021-06-13T00:00:00.000 683, Swanston Street, 3053, Swanston Street, M...
5 1010 arc1112 2022-11-29T17:00:01.000 C 2021-06-29T00:00:00.000 144.95222195 -37.82250002 {'type': 'Point', 'coordinates': [144.95222195... NaN Siddeley Street, 3008, Melbourne, Victoria, Au...
6 1009 arc1050 2022-11-29T17:00:01.000 C 2021-06-14T00:00:00.000 144.96570467 -37.81686763 {'type': 'Point', 'coordinates': [144.96570467... NaN Crumpler, 40-44, Degraves Street, 3000, Degrav...
7 1008 arc1045 NaN R 2021-06-14T00:00:00.000 144.96705703 -37.81746522 {'type': 'Point', 'coordinates': [144.96705703... 2021-06-20T00:00:00.000 Stop 5: Flinders Street Station, Swanston Stre...
8 1014 arc1045 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.967222 -37.8175 {'type': 'Point', 'coordinates': [144.967222, ... NaN Flinders Street, 3000, Melbourne, Victoria, Au...
9 1001 arc1045 NaN R 2019-11-15T00:00:00.000 144.966492 -37.800793 {'type': 'Point', 'coordinates': [144.966492, ... 2021-06-13T00:00:00.000 Lygon Street/Grattan Street, Grattan Street, 3...
10 1013 arc1047 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.956389 -37.811944 {'type': 'Point', 'coordinates': [144.956389, ... NaN William Street, 3000, Melbourne, Victoria, Aus...
11 1003 arc1047 NaN R 2019-11-15T00:00:00.000 144.960923 -37.8023 {'type': 'Point', 'coordinates': [144.960923, ... 2021-06-13T00:00:00.000 Corkman Park (formerly The Corkman Irish Pub),...
12 1007 arc1113 2022-07-20T15:00:01.000 C 2021-05-20T00:00:00.000 144.951835 -37.82246 {'type': 'Point', 'coordinates': [144.951835, ... NaN Siddeley Street, 3008, Melbourne, Victoria, Au...
13 1012 arc1048 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.97 -37.813333 {'type': 'Point', 'coordinates': [144.97, -37.... NaN Bank Australia, 104, Little Collins Street, 30...
14 1015 arc1046 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.9725 -37.810278 {'type': 'Point', 'coordinates': [144.9725, -3... NaN 2, Little Bourke Street, 3000, Little Bourke S...
15 1016 arc1049 2022-11-29T17:00:01.000 C 2021-09-17T00:00:00.000 144.960556 -37.812778 {'type': 'Point', 'coordinates': [144.960556, ... NaN Hardware Lane/Lonsdale Street, Lonsdale Street...
16 EPA Null Null Null Null 144.97 -37.807396 {'type': 'Point', 'coordinates': [-37.8073959,... Null Victoria Street, 3000, Melbourne, Victoria, Au...

Looking at the data above we can see that there are 16 environmental sensors around the city. The geopandas reverse geocode method has provided us with some interesting locations for the sensors, but hopefully they improve the API over time and the addresses become a bit less confusing! Despite the funny names the addresses are still informative. Let's have a look at where the sensors are on a folium map and append their sensor ID number and address.

In [ ]:
# Create (f)igure and base (m)ap with style and zoom level.
f = folium.Figure(width=800, height=600)
m = folium.Map(location=[-37.81368709240999, 144.95738102347036], tiles = 'CartoDB positron', zoom_start=14,  width=800, height=600)

# Feature Group for potential locations layer.
pl = folium.FeatureGroup(name="Sensor Locations")

# Add potential locations and popup information (Location Number, Street View facing proposed area) to Feature Group.
for sensor in sensor_locations.iterrows():
    pl.add_child(
        folium.Marker(
            location = [sensor[1]['latitude'], sensor[1]['longitude']],
            popup = ("<b>Environmental&nbspSensor</b><br>ID:&nbsp;" + str(sensor[1]['site_id'])+"<br>Address:&nbsp"+sensor[1]['Address']),
            icon = folium.Icon(color = 'blue', icon="cloud")
        )
    )

    
# Add potential locations feature group to map.
m.add_child(pl)

# Add layer cocntrol to map.
m.add_child(folium.LayerControl())

#Add map to figure
m.add_to(f)

f
Out[ ]:
Calculating the Outdoor Area of the City of Melbourne

Now that we've had a look at the current sensor network, let's have a look at the area we're interested in monitoring. For a comprehensive air quality study the US EPA recommends establishing air sensors every square metre covering the area of interest. (You can see the discrepancies between that requirement and that of an air monitoring site being 50m away from a road!)

Determining the size of the area we're interested in is therefore a good first step.

First we'll upload the road corridors dataset from the City of Melbourne and calculate the area. We set a limit on the number of records downloaded from the source as otherwise it will stop at 1000. From viewing the dataset on the City of Melbourne website we know that there are 4177 records in total. This ensures they're all captured. Research is extremely important when dealing with data!

In [ ]:
#Loading Melbourne Road Corridor and Footpath Datasets as a Geopandas dataframes.

roaddata = client.get('wzzt-avwf', limit = 10000) 

road_corridors = gpd.GeoDataFrame.from_records(roaddata)

road_corridors.head()
Out[ ]:
gisid seg_id the_geom street_id status_id seg_part str_type dtupdate poly_area seg_descr
0 2145 22837 {'type': 'MultiPolygon', 'coordinates': [[[[14... 2032 2 1 Council Major 20210923 1061 Newman Street between Sambell Street and Frear...
1 1187 22652 {'type': 'MultiPolygon', 'coordinates': [[[[14... 368 2 1 Council Major 20210923 435 Intersection of Wills Street and A'Beckett Street
2 43 30074 {'type': 'MultiPolygon', 'coordinates': [[[[14... 3221 9 1 Rail/Tram 20210923 70196 North Melbourne Railway
3 1120 21410 {'type': 'MultiPolygon', 'coordinates': [[[[14... 761 1 1 Arterial 20210923 1945 King Street between Dudley Street and Walsh St...
4 3928 23269 {'type': 'MultiPolygon', 'coordinates': [[[[14... 818 3 1 Council Minor 20210923 330 Little Errol Street between George Johnson Lan...
In [ ]:
footpathdata = client.get('5che-qtdy',limit = 10000)

footpaths = gpd.GeoDataFrame.from_records(footpathdata)
footpaths.head()
Out[ ]:
the_geom mcc_id str_id ext_id asset_clas asset_type asset_subt name label profile ... addresspt_ addresspt1 easting northing created_us created_da last_edite last_edi_1 shape_star shape_stle
0 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1390340 1390340 RPSP0813103L1 Road Road Footway ... 0 0.0 0.0 0.0 ASSET 2015-05-31T07:00:00.000Z ASSET 2015-05-31T07:00:00.000Z 73.3459618899 67.4163416659
1 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1387134 1387134 RPSP0812119L1 Road Road Footway ... 0 0.0 0.0 0.0 ASSET 2015-05-31T07:00:00.000Z ASSET 2015-05-31T07:00:00.000Z 5.51139959303 9.95058478566
2 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1466439 1466439 RPSP05A10988L1 Road Road Footway ... 0 0.0 0.0 0.0 ASSET 2015-05-31T07:00:00.000Z ASSET 2015-05-31T07:00:00.000Z 14.2438798524 15.5477452293
3 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1390595 1390595 RPSP0911524L1 Road Road Footway ... 0 0.0 0.0 0.0 ASSET 2015-05-31T07:00:00.000Z ASSET 2015-05-31T07:00:00.000Z 12.7903400012 16.863613357
4 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1390205 1390205 RPSP07B10486L1 Road Road Footway ... 0 0.0 0.0 0.0 ASSET 2015-05-31T07:00:00.000Z ASSET 2015-05-31T07:00:00.000Z 52.2056975275 32.9771549323

5 rows × 26 columns

In [ ]:
# Dataset Sources:

# https://data.melbourne.vic.gov.au/Transport/Road-corridors/9mdh-8yau
# https://data.melbourne.vic.gov.au/City-Council/Footpaths/tqjk-32d9

from urllib.request import urlopen
import json

roadsgeoJSON_Id = 'wzzt-avwf'

#Call the API
roadsGeoJSONURL = 'https://'+domain+'/api/geospatial/'+roadsgeoJSON_Id+'?method=export&format=GeoJSON'
with urlopen(roadsGeoJSONURL) as response:
    roadsegments = json.load(response)
    

#Calling the response to observe the JSON file structure.
roadsegments["features"][0]['properties'].keys()
In [ ]:
footpathsgeoJSON_Id = '5che-qtdy'

#Call the API
footpathsGeoJSONURL = 'https://'+domain+'/api/geospatial/'+footpathsgeoJSON_Id+'?method=export&format=GeoJSON'
with urlopen(footpathsGeoJSONURL) as response:
    footpathsegments = json.load(response)    

#Calling the response to observe the JSON file structure.
footpathsegments["features"][0]['properties'].keys()
In [ ]:
#Getting the road data.
dataresource = client.get('wzzt-avwf', limit = 10000) 

# Retrieving data from road corridors dataset. 

road_corridors = gpd.GeoDataFrame.from_records(dataresource)

#Adding a geometry column using shapeleys shape method. This enables the geodataframe to be manipulated and plotted. 
road_corridors['geometry'] = [shape(i) for i in road_corridors['the_geom']]

#Setting coordinate reference system (CRS) for the format of the data (WGS84)
road_corridors.crs = "EPSG:4326"
In [ ]:
#Getting the footpath data.
dataresource = client.get('5che-qtdy', limit = 10000) 

# Retrieve data from footpath corridors dataset.
footpath_corridors = gpd.GeoDataFrame.from_records(dataresource)

#Adding a geometry column using shapeleys shape method. This enables the geodataframe to be manipulated and plotted.
footpath_corridors['geometry'] = [shape(i) for i in footpath_corridors['the_geom']]

#Setting coordinate reference system (CRS) for the format of the data (WGS84)
footpath_corridors.crs = "EPSG:4326"
In [ ]:
style = {'fillColor': '#08af64', 'color': '#08af64'}
style_foot = {'fillColor': '#0f9295', 'color': '#0f9295'}


#Create the base layer map (m)
m = folium.Map(
    location=[-37.81368709240999, 144.95738102347036], #Coordinates are in the Melbourne CBD block
    tiles="cartodbpositron",
    zoom_start=14, 
    control_scale=True,
    prefer_canvas=True, 
    width=800, 
    height=580
)

#Add the geoJSON layer which contains the roads.
folium.GeoJson(roadsGeoJSONURL,
               name="Roads", style_function=lambda x:style,
               tooltip=folium.features.GeoJsonTooltip(fields=['str_type','seg_descr','poly_area'], 
                                                      localize=True)).add_to(m)


#Add the geoJSON layer which contains the footpaths.
folium.GeoJson(footpathsGeoJSONURL,
               name="Footpaths", style_function=lambda x:style_foot,
               tooltip=folium.features.GeoJsonTooltip(fields=['asset_type'], 
                                                      localize=True)).add_to(m)


# Feature Group for potential locations layer.
pl = folium.FeatureGroup(name="Sensor Locations")

# Add potential locations and popup information (Location Number, Street View facing proposed area) to Feature Group.
for sensor in sensor_locations.iterrows():
    pl.add_child(
        folium.Marker(
            location = [sensor[1]['latitude'], sensor[1]['longitude']],
            popup = ("<b>City&nbspof&nbspMelbourne&nbspSensor</b><br>ID:&nbsp;" + str(sensor[1]['site_id'])+"<br>Address:&nbsp"+sensor[1]['Address']),
            icon = folium.Icon(color = 'black', icon="cloud")
        )
    )

    
# Add potential locations feature group to map.
m.add_child(pl)
folium.LayerControl().add_to(m)
#Render the map
m
Out[ ]:
Make this Notebook Trusted to load map: File -> Trust Notebook

By clicking on the layers icon in the upper right of the map you can switch layers on and off.

If we play around with this we can see that the footpath polygons are contained within the road polygons. That's good to know so we don't double up our areas.

In [ ]:
#Creating CBD polygon.
from shapely.geometry import Polygon

lon_point_list = [144.944077, 144.970996, 144.975450, 144.948066]
lat_point_list = [-37.813788,-37.807214,-37.815545,-37.822770]

cbd_geom = Polygon(zip(lon_point_list, lat_point_list))
crs = {'init': 'epsg:4326'}
cbdBoundary = gpd.GeoDataFrame(index=[0], crs=crs, geometry=[cbd_geom])   

cbdBoundary = cbdBoundary.to_crs(epsg=4326)


#Clipping canopy data to only the Melbourne CBD.
cbdRoads = road_corridors.overlay(cbdBoundary, how='intersection')
cbdFootpaths = footpath_corridors.overlay(cbdBoundary, how='intersection')
In [ ]:
#Calling plot on the geodataframes is a good way to check that we've created what we wanted to! The visualisations below show that we've done the right thing.

cbdRoads.plot(color="#0f9295")
cbdFootpaths.plot(color='#0f9295')

cbdRoads.to_file('roads.shp')
# IdealTotalSensors = cbdRoads['poly_area'].sum()
# print("The ideal total number of sensors around the Melbourne CBD is approximately : " + IdealTotalSensors)

WARNING:fiona._env:Value '{"type": "MultiPolygon", "coordinates": [[[[144.961816007658, -37.818937297076], [144.961440487574, -37.81812007738], [144.961503369449, -37.81810172502], [144.961510617506, -37.818117362432], [144.961664512977, -37.818449390885], [144.961644941257, -37.818454943045], [144.961861684516, -37.818922560509], [144.961816007658, -37.818937297076]]]]}' of field the_geom has been truncated to 254 characters.  This warning will not be emitted any more for that layer.
In [ ]:
cbdRoads
Out[ ]:
gisid seg_id the_geom street_id status_id seg_part str_type dtupdate poly_area seg_descr geometry
0 3329 20395 {'type': 'MultiPolygon', 'coordinates': [[[[14... 1116 3 1 Council Minor 20210923 481 Tavistock Place between Flinders Street and Fl... POLYGON ((144.96144 -37.81812, 144.96150 -37.8...
1 3396 20324 {'type': 'MultiPolygon', 'coordinates': [[[[14... 805 3 1 Council Minor 20210923 154 Louden Place from Little Bourke Street POLYGON ((144.96483 -37.81270, 144.96494 -37.8...
2 3294 20432 {'type': 'MultiPolygon', 'coordinates': [[[[14... 697 3 1 Council Minor 20210923 192 Hay Place from Flinders Lane POLYGON ((144.95688 -37.81992, 144.95683 -37.8...
3 3251 20122 {'type': 'MultiPolygon', 'coordinates': [[[[14... 0 2 1 Council Major 20210923 359 Intersection of Exhibition Street and Flinders... POLYGON ((144.97179 -37.81513, 144.97174 -37.8...
4 3590 22551 {'type': 'MultiPolygon', 'coordinates': [[[[14... 780 2 1 Council Major 20210923 1114 La Trobe Street between Adderley Street and Wu... POLYGON ((144.95030 -37.81363, 144.95033 -37.8...
... ... ... ... ... ... ... ... ... ... ... ...
709 3175 20270 {'type': 'MultiPolygon', 'coordinates': [[[[14... 538 3 1 Council Minor 20210923 428 Corrs Lane between Little Bourke Street and Lo... POLYGON ((144.96823 -37.81116, 144.96826 -37.8...
710 3254 20108 {'type': 'MultiPolygon', 'coordinates': [[[[14... 526 2 1 Council Major 20210923 6075 Collins Street between Exhibition Street and R... POLYGON ((144.97021 -37.81427, 144.97099 -37.8...
711 3493 20128 {'type': 'MultiPolygon', 'coordinates': [[[[14... 0 2 1 Council Major 20210923 371 Intersection of Elizabeth Street and Flinders ... POLYGON ((144.96438 -37.81730, 144.96437 -37.8...
712 410 30762 {'type': 'MultiPolygon', 'coordinates': [[[[14... 3224 9 1 Rail/Tram 20210923 1627 Southern Cross Railway POLYGON ((144.95175 -37.81920, 144.95165 -37.8...
713 1166 21514 {'type': 'MultiPolygon', 'coordinates': [[[[14... 740 2 1 Council Major 20210923 4019 Jeffcott Street between Spencer Street and Kin... POLYGON ((144.95155 -37.81203, 144.95114 -37.8...

714 rows × 11 columns

The cbdRoads dataframe has a column called poly_area which states the area of each road segment. It's already in square metres so all we have to do to calculate the total road area for the city of Melbourne is sum this column. To ensure that the sum method works properly we first cast the string to an integer with the .astype(int) method.

In [ ]:
print("The total road area for the Melbourne CBD is " + str(cbdRoads['poly_area'].astype(int).sum()) + " square metres.")   

The total road area for the Melbourne CBD is 1192838 square metres.

If we followed the EPAs advice for air quality studies that would equate to roughly 1.2 million sensors in the CBD alone! Maybe not very realistic. Let's calculate the area of just the footpaths and see what that amounts to.

In [ ]:
cbdFootpaths['area'] = cbdFootpaths['geometry'].area

We can see from the error message that the coordinate reference system we are currently using is a geographic one as opposed to a projected. For calculating distance metrics a projected CRS must be used. The relevant CRS for Melbourne is Map Grid of Australia (MGA) Zone 55. The CRS number is 28355.

In [ ]:
cbdFootpaths['area'] = cbdFootpaths.to_crs(28355)['geometry'].area

#Using to_crs in this line of code doesn't change the default setting for the dataframe. It only uses it for this one calculation.
In [ ]:
footpatharea = cbdFootpaths['area'].astype(int).sum()

print("The total footpath area for the Melbourne CBD is " + str(footpatharea) + " square metres.")

The total footpath area for the Melbourne CBD is 170064 square metres.

So if we only count the area of the footpath areas in the CBD, it is a much more manageable 170k square metres. 170k environmental sensors sounds ok? Maybe not. Multiple studies have been done into the effects of ongoing air pollution on human health. While 170k sensors might sound like a ridiculous target, such a network may produce extremely valuable data, the ramifications of which may not yet be fully understood. Research into the field of large-scale environmental sensor networks is ongoing, and many feasibility studies are being done as technology advances and sensors become ever smaller and cheaper.

As the field of dense environmental sensor networks is still developing, there are few precedents from which to base this use case. There are a host of variables to consider, from the movement speed of different gases to sensor cost.

For the purposes of producing some output, we will consider both the EPA air quality sensor 1m deployment guidelines and a high density sensor network system for air quality studies at Heathrow airport that was deployed between 2011 and 2013. For that study, sensors were deployed every 300m along the perimeter of Heathrow airport. For this use case, we will aim to model one sensor for every 150 square metres of Melbourne CBD footpath.


In [ ]:
print("The total number of environmental sensors required for the Melbourne CBD is " + str(footpatharea/150))

The total number of environmental sensors required for the Melbourne CBD is 1133.76

Our calculations so far tell us that we want approximately 1133 sensors around the cbd. Let's use geopandas to create a network of hypothetical sensors. We can see that there are 1577 entries in the CBDfootpaths dataframe, so for ease of use let's use geopandas centroid method and place a sensor at the centre of each footpath segment.

In [ ]:
idealSensorNetwork = cbdFootpaths.to_crs(28355).centroid
In [ ]:
idealSensorNetwork
Out[ ]:
0       POINT (321537.899 5813159.193)
1       POINT (320702.023 5812703.107)
2       POINT (320947.128 5813166.662)
3       POINT (319896.381 5812890.535)
4       POINT (320921.208 5812755.960)
                     ...              
1572    POINT (321150.804 5812528.278)
1573    POINT (319666.899 5812014.864)
1574    POINT (320479.597 5812596.541)
1575    POINT (319095.160 5812724.660)
1576    POINT (319193.182 5812457.527)
Length: 1577, dtype: geometry

Finally we have our hypothetical sensor network. We'll show each one on the map as a hollow black circle to demonstrate the different options available in Folium.

I hope you've enjoyed this use case and found it both entertaining and informative.

Have a great day!

In [ ]:
# Create a geometry list from the GeoDataFrame
ideal_network_list = [[point.xy[1][0], point.xy[0][0]] for point in idealSensorNetwork.to_crs(4326).geometry ]

f = folium.Figure(width=800, height=600)
m = folium.Map(location=[-37.81368709240999, 144.95738102347036], tiles = 'CartoDB positron', zoom_start=14,  width=800, height=600)

# Iterate through list and add a marker for each sensor.
i = 0
for coordinates in ideal_network_list:
  
    m.add_child(folium.Circle(location = coordinates, radius = 5, color = "#121212", fill = False,
                            popup = ("<b>Proposed&nbspEnvironmental&nbspSensor</b><br>Coordinates: " + str(ideal_network_list[i]))))
                            
    i = i + 1
    
m
Out[ ]:
Make this Notebook Trusted to load map: File -> Trust Notebook

Further Reading:

Low-Cost Outdoor Air Quality Monitoring and Sensor Calibration: A Survey and Critical Analysis

Low-Cost Environmental Sensor Networks: Recent Advances and Future Directions

Systematic Review of Air Quality Sensors

An Integrated Risk Function for Estimating the Global Burden of Disease Attributable to Ambient Fine Particulate Matter Exposure

Intelligent Calibration and Virtual Sensing for Integrated Low-Cost Air Quality Sensors